feat(prometheus): add metrics for AI cache hits, misses, bypasses, and embedding latency - #13659
Conversation
…d embedding latency
dosubot
Bot
added
size:XL
| metrics[name]:inc(1, | ||
| get_enabled_label_values_for_metric(name, disabled_label_metric_map, | ||
| layer or "exact", route_id, service_id, consumer_name, balancer_ip, | ||
| vars.request_type, vars.request_llm_model, vars.llm_model, |
There was a problem hiding this comment.
Refer to this PR: #13637
As the model name is from Client, I suggest we should also limit the Maximum length 🤔
| | route_id | ID of the Route that the metric corresponds to. Default to an empty string if a request does not match any Route. | | ||
| | service_id | ID of the Service that the matched Route belongs to. Default to an empty string if the matched Route does not belong to any Service. | | ||
| | consumer | Name of the Consumer associated with a request. Default to an empty string if no Consumer is associated with the request. | | ||
| | node | IP address of the upstream node. Empty for requests served from the cache, which never reach the upstream. | |
There was a problem hiding this comment.
Same problem as below?
| | node | IP address of the upstream node. Empty for requests served from the cache, which never reach the upstream. | | ||
| | request_type | traditional_http / ai_chat / ai_stream | | ||
| | request_llm_model | Model name requested by the client. | | ||
| | llm_model | Model name reported by the LLM response. Empty for requests served from the cache, which never reach the upstream. | |
There was a problem hiding this comment.
How to understand "Empty for requests served from the cache, which never reach the upstream. "?
There was a problem hiding this comment.
Since the requests is served from cache, meaning we never called the upstream to get the response back hence empty. Should i reword this or still shows the model neverthelesss?
dosubot
Bot
added
size:L
membphis
left a comment
membphis
left a comment
There was a problem hiding this comment.
LGTM. No merge-blocking issues found in the review.
| if ctx.ai_cache_status then | ||
| exporter.inc_ai_cache_status(ctx, ctx.ai_cache_status, ctx.ai_cache_hit_layer) | ||
| end | ||
| if ctx.ai_cache_embedding_latency then | ||
| exporter.observe_ai_cache_embedding_latency(ctx, ctx.ai_cache_embedding_latency) | ||
| end |
There was a problem hiding this comment.
all values passed to these functions are in the ctx table, so why can't we directly call these functions in the exporter module directly?
There was a problem hiding this comment.
agreed, that'd be cleaner. fixed
… and update test configuration
membphis
left a comment
membphis
left a comment
There was a problem hiding this comment.
Code review passed; no blocking issues found.
5 participants
Description
Adds Prometheus metrics for the
ai-cacheplugin (#13578, #13632, #13644), integrated inprometheus/exporter.luafollowing the existingllm_*metric pattern:apisix_ai_cache_hits_total(withlayer="exact"|"semantic"),apisix_ai_cache_misses_total,apisix_ai_cache_bypasses_total— counters sharing thellm_*label set (route_id,service_id,consumer,node,request_type,request_llm_model,llm_model)apisix_ai_cache_embedding_latency— histogram of embedding-call latency in milliseconds (the issue sketched_seconds, but the exporter's latency histograms use milliseconds withDEFAULT_BUCKETS, so this follows the house convention)Semantics: Redis fail-open lookups count as MISS;
fail_mode: errorrejections record nothing; recording happens inlogphase and is a silent no-op when theprometheusplugin is disabled. Docs: label sections added toprometheus.md(en + zh).Which issue(s) this PR fixes:
Fixes #13290
Checklist